Image synthesizing device and method

ABSTRACT

A one-dimensional image synthesizing unit  3  images an subject, and produces a plurality of image-to-be-combined signals expressing images-to-be-combined having overlapped regions. A central processing unit  5  first obtains matching of images of the subject for each of adjacent images-to-be-combined for all image-to-be-combined signals by center and large block matching processes, and detects the deviations of overlapped regions from the obtained matching. Next, in a small block matching process, on the basis of the deviations of the overlapped regions, a plurality of reference blocks and search regions are made to correspond to each other and are set in the respective overlapped regions, and the deviations of the images of the subject are detected in a unit of each search region in more detail than the deviations detected by the large block matching process. Finally, each of the images-to-be-combined is partially geometrically deformed so as to cancel the deviation of the image of the subject, the reference block and the search block are made coincident with each other, and all image-to-be-combined signals are combined to superimpose the respective images-to-be-combined, so that combined image signals are obtained.

TECHNICAL FIELD

The present invention relates to an image synthesizing apparatus andmethod for obtaining an image which has the number of pixels larger thanan image obtained by only one imaging operation using an image pickupdevice and which has high resolution, or has a wide angle of view andwide range.

BACKGROUND ART

In recent years, in an image synthesizing apparatus used as a so-calledscanner, a one-dimensional image pickup device in which light receivingareas are linearly arranged, such as a CCD line sensor, is used. When anoutput image is obtained by this image synthesizing apparatus, first,while the image pickup device is moved relative to a subject on atwo-dimensional plane in the direction perpendicular to the arrangementdirection of the light receiving areas, an image of light from thesubject is taken at a plurality of different positions. Next, the imagesobtained in the respective imaging operations are combined on thetwo-dimensional plane so that they are arranged with the same positionalrelation as the position of the image pickup device at the respectiveimaging operations, and the output image is obtained.

In Japanese Unexamined Patent Publication JP-A 5-260264, the presentapplicant has proposed a technique which uses a two-dimensional imagepickup device with light receiving areas arranged in matrix and obtainsan output image composed of pixels the number of which is larger thanthe number of the light receiving areas of the image pickup device. Inan image processing apparatus of the publication, an operator of theapparatus takes an image of light while horizontally moving an imageinput means of a so-called CCD camera so that a plurality of images areobtained. At every time when a new image is obtained, a processingcircuit first makes a check on matching of the new image and a combinedimage, and obtains a difference between these images. Next, on the basisof the obtained difference, a new image is deformed, and finally, theimage after deformation and the combined image are combined, so that anew combined image is obtained.

With respect to the image obtained by the two apparatuses describedabove, as compared with an image directly obtained from an image pickupdevice, although the number of pixels arranged in the direction alongthe moving direction of the image pickup device is increased, the numberof pixels in the direction orthogonal to the moving direction is notchanged. As a technique for increasing the number of pixels in theorthogonal direction of the image, there is a technique in which theplurality of output images are arranged along the orthogonal direction,and are further combined.

Japanese Unexamined Patent Publication JP-A 63-64180 discloses an imagesynthesizing method of the technique mentioned above. The apparatus ofthe picture image synthesizing method uses a hand scanner which isprovided with a one-dimensional image pickup device and can be manuallymoved. In the method, first, an original image of an input subject isdivided into regions having an appropriate size to read by one scanningusing the hand scanner, and image pickup is carried out while the handscanner is moved along the center axis of the regions in the state thatthe hand scanner is made to be in contact with the original image, sothat images of the respective regions are obtained. The image of each ofthe regions is equivalent to the output image of the apparatus using theone-dimensional image pickup device, and includes an overlapped regionin which the same portion of the original image is read. Next, a checkon matching of the overlapped regions of the images in the respectiveregions is carried out, and regions in the images where images of thesubject are coincident with each other are obtained. Finally,superimposition is made after translation of the respective images iscarried out to superimpose these images, so that a combined image of apicture image is obtained.

In the picture image synthesizing method described above, since the handscanner including the image pickup device is manually moved by theoperator of the apparatus, at the movement, there is a case where anunintentional movement of the operator's hands is applied to the handscanner. Moreover, from the same reason, there is a case where therelative position between the image pickup device in the hand scannerand the center axis of the regions of the original image of the inputsubject is tilted. Moreover, although the hand scanner is provided witha pair of rollers at a portion which is brought into contact with anoriginal image so that the hand scanner smoothly moves on the surface ofthe original image, when the smoothness of one of the rollers isdifferent from that of the other, there is a case where the moving speedof the hand scanner is different between a portion in the vicinity ofone roller and a portion in the vicinity of the other roller.

When these happen, in the image obtained by the apparatus, distortion ofan image, such as deviation of an image of the subject and partialcompression of the image, occurs. In the image in which such distortionoccurred, images of the subject of a plurality of images are notsmoothly connected by only the horizontal movement of the images, sothat distortion occurs in the image of the subject in the combinedimage.

In the image processing apparatus of JP-A 5-260264, the image inputmeans takes image light of an subject from a position apart from thesubject by a predetermined distant. This image input means is supportedby only a hand of an operator of the apparatus and is held in the air,so that the relative position to the subject is apt to be shifted. Whenthe relative position is shifted like this, distortion of an image, suchas deviation, tilt, and change in magnification of an image, occurs inthe obtained images of the subject in the images-to-be-combined.Although the apparatus combines the images while correcting thedistortions of the images of the subject in combining the images, thereis a case where the distortion remains since all distortions of theimage can not be corrected by, for example, an error in matching and acorrection error.

An object of the invention is to provide an image synthesizing apparatusand method capable of obtaining an image in which an image of a subjecthas a small distortion, and which has high resolution or a wide angle ofview and wide range, by using an image pickup device with a small numberof light receiving areas.

DISCLOSURE OF THE INVENTION

The present invention provides an image synthesizing apparatuscomprising:

image-to-be-combined producing means for producing a plurality ofsignals of images-to-be-combined, the signals representing theimages-to-be-combined including different portions of a subject, themeans producing signals so that an image-to-be-combined including aportion of the subject and an adjacent image-to-be-combined including aportion adjacent to said portion of the subject have an overlappedregion where a same portion of the subject is imaged in theimages-to-be-combined;

first matching means for making a check on matching of the images of thesubject in the overlapped region for each set of adjacentimages-to-be-combined among the image-to-be-combined signals produced bythe image-to-be-combined producing means, and detecting a relativepositional deviation between the overlapped regions on the basis of aresult of the check on matching;

second matching means for setting a plurality of reference regions of apredetermined size in the overlapped region of one of theimages-to-be-combined whose relative positional deviation is detected bythe first matching means, for setting a search region larger than thereference region for each of the reference regions in the overlappedregion of the other of the images-to-be-combined on the basis of therelative positional deviation, for making a check on matching of theimages of the subject between the reference region and the search regioncorresponding to the reference region, and for detecting a relativepositional deviation between the images of the subject on the basis of aresult of the check on matching; and

image combining means for combining all the image-to-be-combined signalsto superimpose the reference region and the search region correspondingto the reference region for each set of adjacent images-to-be-combinedwhile partially deforming the respective images-to-be-combined to cancelthe relative positional deviation detected by the second matching means,and producing combined image signals expressing a single combined imagewith respect to the subject.

According to the invention, the image synthesizing apparatus combines aplurality of images-to-be-combined so that the images of the subject inthe overlapped regions are coincident with each other and a singlecombined image is obtained. The images-to-be-combined are originalimages imaged by an image pickup apparatus such as a video camera.

This image pickup apparatus takes image light from finite regions of oneand the same subject while horizontally and vertically moving the finiteregion which can be imaged by one imaging operation of the image pickupapparatus, so that the original images to become theimages-to-be-combined are obtained. At this time, in the obtainedoriginal images, when the positions of the images of a portion of thesubject are different, in spite of the fact that the same subject isimaged, there is a case where a difference occurs in the shape of theimages of the subject by, for example, distortion of a lens of anoptical system of the image pickup apparatus. For example, when theimage pickup apparatus is held by a hand of an operator of the imagesynthesizing apparatus and is moved, the relative distance between thesubject and the image pickup apparatus is changed in imaging, so thatthere is a case where a difference occurs in the size of the images ofthe subject in the original images. Further, when theimage-to-be-combined is a combined image in which the plurality oforiginal images are combined, there is a case where the difference ofthe images of the subject in the respective original images isaccumulated, and the difference becomes large.

In the image combining method of the image synthesizing apparatus,first, a result of the check on matching as an index expressing thedeviations of the images of the subject in the overlapped regions of therespective images-to-be-combined is obtained. This result of the checkon matching indicates the deviation of position of the images of thesubject in the overlapped regions and the degree of deformation of theshape of the images of the subject for every set of adjacentimages-to-be-combined, for example. Next, from the obtained result ofthe check on matching, the relative positional deviation the images ofthe subject is detected. Finally, in order to correct the detecteddeviation and deformation, the respective reference regions in therespective overlapped regions and the search regions correspondingthereto are superimposed so that they are coincident with each other,and all the images-to-be-combined are combined while deforming therespective images-to-be-combined.

By this, the difference is corrected, and it is possible to obtain acombined image with a wider angle of view and wider range than theimage-to-be-combined or a combined image with high resolution. As amethod of canceling the deviation, not only the relative translation ofimages-to-be-combined of adjacent set but also deformation of images iscarried out, so that it is also possible to correct, for example, achange in partial magnification of the image pickup apparatus inproducing the images-to-be-combined.

The matching process for detecting the deviation of the subject iscarried out in two steps. Specifically, in the first matching process inthe first matching means, the deviation of the subject to determine anamount of translation of the overlapped regions is roughly detected. Inthe second matching process in the second matching means, to determine adeformation amount of deformation of the images, an amount of deviationof the subject is detected in more detail than the first. Thus, thematching process in the first matching means is, for example, a similarprocess to the second matching means, and the reference region can beset larger than the reference region of the second matching means. Bythis, an amount of processing of the first matching means can bedecreased.

Further, in the matching process operation of the second matching means,on the basis of the detection result of the first matching means, forexample, the position where the best matching is obtained in theimages-to-be-combined is roughly estimated, and the position of thesearch region to be searched is set to include that position. Thus, ascompared with the case where the search region is set mechanically atrandom in the image-to-be-combined, even if the size of the searchregion is made small, the position of the image of the subject to besearched is easily made coincident with the position of the searchregion. Thus, it is possible to decrease the search region and todecrease the processing amount of the second matching means.

Further, in the invention it is preferable that the first matchingmeans:

firstly, sets a first reference region of a predetermined size at apredetermined position in an overlapping area of one of the set of theadjacent images-to-be-combined, making a check on matching of the imagesof the subject between the overlapped region of the otherimage-to-be-combined and the first reference region, and detects arelative positional deviation of the images of the subject on the basisof the obtained matching;

next, sets at least two second reference regions smaller than the firstreference region in parallel with the detected deviation direction inthe one of the overlapped regions, and sets a second search regionsmaller than the overlapped region in the overlapped region of the othersubject region for each of the second reference regions on the basis ofthe detected relative positional deviation; and

makes a check on matching of the images of the subject between thesecond search region and the second reference region corresponding tothe second search region, and detects the relative positional deviationbetween the overlapped regions on the basis of an obtained result of thecheck on matching.

According to the invention, the matching process in the first matchingmeans is carried out in two steps. Although a plurality ofimages-to-be-combined with respect to an subject are produced by theimage-to-be-combined production means, it is difficult to judge inimaging how the relative position of images of the subject in a pair ofadjacent images-to-be-combined to be arranged in parallel is shifted.For example, it is difficult to judge in imaging whether, in a pair ofimages-to-be-combined superimposed and arranged horizontally so that theimages of the subject in the overlapped regions are coincident with eachother, the upper side of the right image-to-be-combined relativelyprotrudes to the upper side of the left image-to-be-combined, or whetherthe upper side of the left image-to-be-combined relatively protrudes tothe upper side of the right image-to-be-combined. When this deviation isnot known, it become difficult to determine the optimum position in theoverlapped regions where the reference region and the search region fordetecting the relative positional deviation between the overlappedregions are to be set.

In the first matching means of the invention, first, by using the firstreference region, the deviation of the images of the subject between apair of images-to-be-combined is detected, and on the basis of thedetected deviation, places where the second reference region and thesecond search region can be arranged are determined. Next, the secondreference region and the second search region are set at the placeswhere arrangement can be made, and the deviations of the images of thesubject between the overlapped regions are obtained. By carrying out thematching process in two steps in this way, even in the case where thestate of the deviation of the pair of images-to-be-combined isindefinite, the second reference region and the second search region inthe second matching process can be positioned at the optimum place fordetecting the relative positional deviation between the overlappedregions.

Since the deviation between the overlapped regions is roughly detectedin the first matching process, the search region of the second matchingprocess can be made smaller than the overlapped region. Further, in thesecond matching process, results of the check on matching are obtainedat two places in the overlapped regions, so that not only a paralleldeviation between the images of the subjects but also a change in sizeand a deviation in rotation of the images can be detected.

In the matching process, although the accuracy in matching can beimproved as the reference region is made small, a possibility that apoint which does not correspond to an image of the subject in thereference region is erroneously regarded as a coincident point in theregion to be searched, is increased. Thus, it is possible to prevent theoccurrence of error by making the second reference region of the secondmatching process larger than the first reference region of the firstmatching process.

Further, in the invention it is preferable that the image synthesizingapparatus further comprises rotation transformation means for obtaininga relative rotation angle between the adjacent images-to-be-combined foreach set of the adjacent images-to-be-combined on the basis of therelative positional deviation between the overlapped regions detected bythe first matching means, and for making rotation transformation of theimage-to-be-combined signals so that the respectiveimages-to-be-combined undergo relative angular displacement in thedirection where the obtained relative rotation angle is cancelled; and

the second matching means makes a check on matching of the search regionfor set of the images-to-be-combined in which the relative position isrotatively transformed by the rotation transformation means.

According to the invention, the image synthesizing apparatus previouslycorrects, by the rotation transformation, only the deviation of rotationmovement expressed by the relative rotation angle of the relativepositional deviation for each pair of images-to-be-combined after thematching process in the first matching means, and applies the matchingprocess in the second matching means to the correctedimages-to-be-combined. By combining the images-to-be-combined in whichsuch rotation correction has been carried out, it is possible to preventthe occurrence of deviation and distortion due to the rotation movementof the images in the combined image and to smoothly connect the images.

In the matching process by the second matching means, basically, it isappropriate that only an amount of translation in the deviations of theimages of the subject at a plurality of positions in the overlappedregions is detected. Since the amount of translation is once detected bythe first matching means, the process amount of the matching process canbe decreased by further decreasing the search region of the matchingprocess of the second matching means. Even in the case where thedeviation of rotation movement remains after the rotationtransformation, the deviation is minute, so that even if the searchregion is made small, detection can be made.

Further, in the invention it is preferable that the second matchingmeans gives to the image synthesizing means only a result of the checkon matching for the search region in which the image of the subjectmeets a predetermined index of eligibility as a comparative objective,among results on matching obtained for the respective plurality ofsearch regions.

According to the invention, in the image synthesizing method, among theplurality of results of the check on matching obtained by the secondmatching means, for example, only a high reliable matching is used foran image deformation process in the image synthesizing means. A highreliable result on matching can be obtained when the image of thesubject in the search region to be set meets a predetermined index ofeligibility as a comparative objective. As the index of the eligibilityas a comparative objective, for example, with reference to theprocessing method of the matching process, it is preferable to select anindex which can be detected quantitatively. By this, it is possible toprevent such a case that a combining process is carried out using anerroneous result on matching and the distortion of images of the subjectin the combined image is increased.

Further, in the invention it is preferable that each of theimages-to-be-combined is constituted by a plurality of pixels;

the reference region is equal to or larger than a single pixel in size;

the result on matching to the respective search regions is expressed ina position of the search range which is equal to the reference region insize and shape in the respective search regions, and in which a totalsum value of pixel values expressing luminance of all pixels in thesearch range is closest to a total sum of pixel values of the referenceregion; and

it indicates eligibility of the objectives as a comparative objectivethat a difference value between the total sum value of the pixel valuesin the search range of the check on matching and the total sum value ofthe pixel values of the reference region is less than a predeterminedreference value.

According to the invention, in the second matching means, a result ofthe check on matching is obtained by a processing method called aso-called template matching. At this time, a difference value betweenthe total sum value of the pixel values of all pixels in the searchrange and the total sum value of the pixel values of the referenceregion shadows eligibility as a comparative objective. When thedifference value is not less than a predetermined reference value,although the image of the subject in the search range obtained as aresult on matching is closest to the image of the subject in thereference region in the search region, a possibility that it is an imageof a portion different from the image of the reference region in thesubject becomes high. Thus, it is preferable to remove the matchingexpressing the position of such search range.

Further, in the invention it is preferable that each of theimages-to-be-combined is constituted by a plurality of pixels;

the reference region is equal to or larger than a single pixel in size;

the result of the check on matching to the respective search regions isexpressed in a position of a search range which is equal to thereference region in size and shape in the respective search regions, andin which a total sum value of pixel values expressing luminance of allpixels in the search range is closest to a total sum value of pixelvalues of the reference region; and

it indicates eligibly of the objective as a comparative objective that adifference value between a maximum value and a minimum value among pixelvalues in the search range of the matching is not less than apredetermined reference value.

According to the invention, in the second matching means, a result ofthe check on matching is obtained by a processing method called aso-called template matching. At this time, the eligibility of theobjectives as a comparative objective is indicated by the differencevalue between the maximum value and the minimum value in the pixelvalues of all pixels of the search region. When the difference value isless than the predetermined reference value, it is conceivable that theimage in the search region is an image with little change in luminance,such as a white image. Even if a plurality of search ranges are set atdifferent places in such a search region, since images in the respectivesearch ranges are almost the same, it becomes difficult to obtain anaccurate result of the check on matching. Thus, it is preferable toremove a matching in such a search region.

Further, in the invention it is preferable that it indicates eligibilityof the objectives as a comparative objective that in the images of thesubject in the search region, portions with the same change in luminanceand the same shape are not periodically repeated plural times.

According to the invention, the eligibility of the objectives as acomparative objective is determined in such a manner that an image ofthe subject having the periodicity is removed. As such an image of thesubject, for example, there are enumerated a straight line and a patternin which a plurality of images with some pattern are arranged. When theimage in the search region has such periodicity, images in the pluralityof ranges in the plurality of search ranges become equal to each other,so that it becomes difficult to judge which search range in these is thesame portion as the image of the subject of the reference region. Thus,it is preferable to remove the matching in such a search region.

Further, in the invention it is preferable that the image-to-be-combinedis formed of a plurality of pixels arranged in a matrix form alongpredetermined one and another direction; and

in the geometrical transformation, for each of theimages-to-be-combined, a distance between a predetermined transformationreference axis substantially in parallel with the one direction and eachof the pixels is enlarged or reduced in the overlapped region, and amutual distance in a pixel group composed of a plurality of pixelsarranged linearly along the other direction is enlarged or reduced, sothat a pixel arrangement is transformed.

According to the invention, in the geometrical transformation method ofthe image synthesizing means, as described above, the transformationreference axis is fixed, and trapezoid transformation is carried out sothat, for example, a rectangular image is deformed into a trapezoidimage. For example, in the geometrical transformation, when a method inwhich only the translation of the pixel group is carried out accordingto the deviation of the image of the subject, is used, the deviation isaccumulated, and there is a case where distortion occurs in the image ofthe subject in the entire of the combined image. This distortion appearsespecially in the case where not less than three images-to-be-combinedare combined. If the trapezoid transformation is carried out, thedeviation is cancelled by a unit of pixel group in the respectiveimages-to-be-combined, so that it does not affect otherimages-to-be-combined. Thus, by the method similar to the translation,the accumulation of deviation can be prevented. Since the deviation iscancelled in units of pixel group, among the arrangement of matrix ofpixels, with respect to either one of the row and column, it is possibleto keep arrangement relation other than a distance of the respectiverows or columns and a pixel interval of single row or column.

Further, in the invention it is preferable that the transformationreference axis passes through the centers of the respective referenceregions of the second matching means.

According to the invention, with respect to a plurality of pixelsarranged on the transformation reference axis at least beforetransformation, even if an interval between pixels adjacent in onedirection is changed, they are positioned on the transformationreference axis even after transformation. When the plurality ofreference regions of the second matching means are set on thetransformation axis, in the case where not less than threeimages-to-be-combined are combined, and after the images-to-be-combinedof adjacent set are combined, when the image of the combined result isfurther combined with another image-to-be-combined or the image of thecombined result, since the deviations of the reference regions in theother direction do not exist, the processing amount of calculationprocess is decreased. Thus, the processing amount of geometricalcombining process is decreased.

Further, in the invention it is preferable that the image-to-be-combinedand the combined image are respectively composed of a plurality ofpixels; and

among all pixels of the combined image, a pixel value expressingluminance of each pixel of a portion corresponding to the overlappedregion of the image-to-be-combined is a weighted average value obtainedby attaching a weight, which is determined corresponding to a distancefrom a predetermined combining reference axis in the overlapped region,to a pixel value of each pixel of two overlapped regions of theimage-to-be-combined corresponding to the pixel.

According to the invention, among the combined images obtained by theimage synthesizing apparatus, the portion corresponding to theoverlapped portion of the respective images-to-be-combined is formed bysuperimposing the images-to-be-combined of adjacent set. The pixel valueof this portion is obtained by weighted average of pixel value of eachpixel of the two superimposed overlapped regions. By this, in the pairof images-to-be-combined, even when the luminance of the entire of oneor the other image-to-be-combined is different, it is possible tosmoothly change the luminance of the image of the subject from one imageto the other image. Thus, the difference in luminance does not becomenoticeable.

Further, the invention provides an image synthesizing method comprisingthe steps of:

producing a plurality of signals of images-to-be-combined, the signalsrepresenting the images-to-be-combined including different portions of asubject, so that an image-to-be-combined including a portion of thesubject and an adjacent image-to-be-combined including a portionadjacent to said portion of the subject have an overlapped region wherea same portion of the subject is imaged in the images-to-be-combined;

making a check on matching of the images of the subject in theoverlapped region for each set of adjacent images-to-be-combined amongthe produced image-to-be-combined signals, and detecting a relativepositional deviation between the overlapped regions on the basis of aresult of the check on matching;

setting a plurality of reference regions of a predetermined size in theoverlapped region of one of the images-to-be-combined whose relativepositional deviation is detected, for setting a search region largerthan the reference region for each of the reference regions in theoverlapped region of the other of the images-to-be-combined on the basisof the relative positional deviation, for making a check on matching ofthe images of the subject between the reference region and the searchregion corresponding to the reference region, and for detecting arelative positional deviation between the images of the subject on thebasis of a result of the check on matching; and

combining all the image-to-be-combined signals to superimpose thereference region and the search region corresponding to the referenceregion for each set of adjacent images-to-be-combined while partiallydeforming the respective images-to-be-combined to cancel the detectedrelative positional deviation, and producing combined image signalsexpressing a single combined image with respect to the subject.

According to the invention, the image synthesizing method detects thedeviations of the subject through the two-step checking process onmatching by the image synthesizing apparatus. By this, the number ofmatching processes is decreased, and a combined image with highresolution or a wide range and wide angle of view can be obtained.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram for showing an electrical structure of animage synthesizing apparatus 1 of an embodiment of the presentinvention.

FIG. 2 is a schematic view showing the relation between a subject 11 andimage pickup regions 13 to 15.

FIG. 3 is a schematic view showing images-to-be-combined 17 to 19.

FIG. 4 is a schematic view for explaining a method of moving a videocamera 28 of a one-dimensional image synthesizing unit 3.

FIG. 5A and FIG. 5B are views showing original images 37 to 40 obtainedby the video camera 28 and a schematic view for explaining a method ofcombining the original images 37 to 40.

FIG. 6 is a schematic view for explaining a method of moving a handscanner 48 of the one-dimensional image synthesizing unit 3.

FIG. 7 is a flowchart for explaining an image synthesizing method usingthe image synthesizing apparatus 1.

FIG. 8 is a schematic view showing the relation among reference block K,search region 55, and search block L at a center block matching process.

FIG. 9 is a schematic view showing the state in which with respect to apair of images-to-be-combined 17 and 18, the reference block K andminimum search block La at the center block matching process are made toagree with each other.

FIG. 10 is a schematic view showing the relation among reference blocksf1, f2, search regions 58, 59, and minimum search blocks g1 a, g2 a at alarge block matching process.

FIG. 11 is a flowchart for explaining a small block matching processmethod.

FIG. 12 is a schematic view showing the relation among reference blocksba1 to ba12, search region 61, and search blocks bb1 to bb12 at thesmall block matching process.

FIG. 13 is a schematic view showing the relation among reference blocksba1 to ba12, and minimum search blocks bba1 to bba12 at the small blockmatching process.

FIG. 14 is a schematic view of the images-to-be-combined 17 and 18 forexplaining a deformation process of images.

FIG. 15 is a graph showing the relation between a coefficient value ofweighted average of pixel values and a spatial position of pixels in acombining process of images.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 is a block diagram showing an electrical structure of an imagesynthesizing apparatus 1 of an embodiment of the present invention. Theimage synthesizing apparatus 1 includes a one-dimensional imagesynthesizing unit 3, a memory 4, a central processing unit 5, and anoutput unit 6. These structural members 3 to 6 are electricallyconnected to each other through a bus line 7 to give and receivesignals.

As described later, the one-dimensional image synthesizing unit 3individually and continuously images a subject, and outputs a pluralityof signals expressing images-to-be-combined. The image-to-be-combined isa so-called gray scale image in which luminance is changed stepwise fromwhite to black. The plurality of signals of images-to-be-combined arestored in the memory 4. The central processing unit 5 reads theplurality of signals of images-to-be-combined stored in the memory 4 andcombines them by a combining method described later, so that a combinedimage signal expressing a single combined image is produced. Thiscombined image signal is stored in the memory 4, and is given to theoutput unit 6 which is realized by a cathode ray tube (CRT) and aprinter. The output unit 6 visually displays the combined image signalas a combined image.

FIG. 2 is a schematic view showing a subject 11. In this embodiment, thesubject 11 is a flat plate-like member whose surface is imaged. Ofcourse, the subject 11 is not limited to one with this shape, but anyshape may be adopted as long as it can be imaged by an image pickupapparatus such as a still camera and a video camera. The one-dimensionalimage synthesizing unit 3 images the surface of the subject 11 tovirtually divide the surface into a plurality of image pickup regions.In this embodiment, the surface of the subject 11 is individually imagedas three rectangular image pickup regions 13 to 15 disposed in parallelwith each other. At this time, the image pickup regions 13 to 15 are setso that adjacent image pickup regions have an overlapped region.

FIG. 3 is a view showing images-to-be-combined 17 to 19 obtained byimaging the image pickup regions 13 to 15. The images-to-be-combined 17to 19 include overlapped regions 21; 22, 23; 24. The overlapped region21 of the image-to-be-combined 17 and the overlapped region 22 of theimage-to-be-combined 18 have an image of a part 25 of the subject 11.Similarly, the overlapped region 23 of the image-to-be-combined 18 andthe overlapped region 24 of the image-to-be-combined 19 have an image ofa part 26 of the subject 11. The parts 25 and 26 are overlapped portionsof the above-mentioned image pickup regions 13 to 15, and are shown withoblique lines in FIG. 2. It is preferable that theseimages-to-be-combined 17 to 18 have a low parallax in imaging. As animaging method with a low parallax, imaging of horizontal and verticalmovement called panning and tilting may be cited.

FIG. 4 is a view for explaining a specific imaging operation of theone-dimensional image synthesizing unit 3. The one-dimensional imagesynthesizing unit 3 includes, for example, a video camera 28 equippedwith a so-called CCD area sensor, as an image pickup apparatus. The CCDarea sensor is a two-dimensional image pickup device in which lightreceiving areas are arranged in matrix on an imaging surface as atwo-dimensional plane.

The start and end of the imaging operation of the video camera 28 isdetermined by, for example, an instruction by an operator of the imagesynthesizing apparatus 1. When the video camera 28 starts to image thesubject 11, the operator holds the video camera 28 with his hands andmoves it at a position apart from the surface of the subject 11 by apredetermined distance, so that an image pickup feasible range 29 movesaccording to the image pickup regions 13 to 15. This moving operation iscarried out specifically such that the operator sees a finder of thevideo camera 28 or an image in the display device visually displayingthe presently obtained image, determines a moving direction and movingamount of the range 29 so that the image pickup feasible range 29includes a part of a region in which image pickup has been ended, andmoves the video camera 28 so that the range 29 is moved in the movingdirection and by the moving amount. This movement of the video camera 28may be carried out by using a mechanical moving system, other than themanual operation by an operator, as long as the condition describedabove is satisfied.

By this, for example, when FIG. 4 is used as an example, the imagepickup feasible range 29 is moved in the direction indicated by an arrow32 so that its center point 30 first moves from one end to the other endof the image pickup region 13 in the longitudinal direction along thecenter axis in the longitudinal direction. At this time, the width ofthe image pickup region 13 in its width direction is equal to the widthof the image pickup feasible range 30 in its longitudinal direction, andthe image pickup feasible range 30 is moved so as to keep a state thatits center axis passing through the center point 30 is in parallel withthe center axis of the image pickup region 13.

Next, the image pickup feasible range 29 is horizontally moved in thedirection indicated by an arrow 33 so that the image pickup feasiblerange 29 reaches the other end of the image pickup region 14 in itslongitudinal direction. Subsequently, the center point 30 of the imagepickup feasible range 29 is moved from the other end to one end of theimage pickup region 14 in the longitudinal direction in the directionindicated by an arrow 34 along the center axis in the longitudinaldirection. Next, the image pickup feasible range 29 is horizontallymoved in the direction indicated by an arrow 33 so that the image pickupfeasible range 29 reaches one end of the image pickup region 15, andafter the movement, it is moved in almost the same way as imaging in theimage pickup region 13. By this, the image pickup feasible range 29 ismoved so that the center point 30 draws a locus 35.

After the operator's instruction to start image pickup, the video camera28, for example, concentrates light so that image light from the subjectin the image pickup feasible range 29 makes an image on the imagingsurface of the image pickup device, and is received by the respectivelight receiving areas at the imaging surface. At each time when apredetermined light exposure time elapses, the image pickup deviceoutputs electric charges corresponding to the amount of received lightin the light exposure time as pixel data to the respective lightreceiving areas. By this, the video camera 28 outputs an original imagesignal composed of the pixel data at each time when the light exposuretime elapses. The original image signal expresses an original image inwhich the number of pixels is equal to the number of light receivingareas of the image pickup device at the imaging surface and thearrangement of the pixels is similar to the arrangement of the lightreceiving areas. This original image is an image of a so-called oneframe of moving images obtained by the video camera. Thus, while theimage pickup feasible range 29 is moved, the video camera 28 outputsoriginal image signals of plural frames.

For example, when the image pickup feasible range 29 is moved along theimage pickup region 13, it is assumed that original image signals forfour frames are continuously obtained. FIG. 5A and FIG. 5B are viewsshowing original images 37 to 40 expressed by the original image signalsin the order of imaging. The original images 37 to 40 include images 41to 44 of the subject in which the same subject is imaged at differentpositions. These images 41 to 44 are coincident with each other whenparts obtained by imaging the same part of the subject are superimposed,except cases where, for example, the subject moves during the imagingoperation, and the parallax of the video camera 28 and magnification ofan optical system change during the imaging operation.

The video cameral 28 detects correlation of the images of the subject ofthe original images of the respective frames obtained by the imagingoperation, and from the detected correlation, the moving direction ofthe relative position between the image pickup feasible range 29 and thesubject is detected. When this moving direction is turned, for example,when it is turned from the direction of the arrows 32 and 34 to thedirection of the arrow 33, it is judged that imaging of the image pickupregions 13 and 14 is ended, and it is moved to the next image pickupregions 14 and 15. By this, even when the image pickup regions 13 to 15are continuously imaged, the original images obtained by imaging therespective image pickup regions 13 to 15 can be automatically selected.

The relative position between the images of the subject in theseoriginal images 37 to 40 and the frame line of the original images 37 to40 shifts not only in the original moving direction of the image pickupfeasible range 29 indicated by the arrow 32 but also in the directionindicated by the arrow 33 and in the direction opposite thereto. Itappears that this occurs because when the video camera 28 is moved bythe method described above, the video camera 28 is supported by only thehand of the operator, so that the relative position to the subject 11can shift by, for example, an unintentional movement of the operator'shands.

The one-dimensional image synthesizing unit 3 combines the originalimage signals of plural frames, and produces a signal of theabove-mentioned image-to-be-combined 17. The method of producing signalsof the images-to-be-combined 18 and 19 is also equal to that of theimage-to-be-combined 17. The image-to-be combined is an image in whichthe original images 37 to 40 are superimposed so that theimages-to-be-combined agree with each other.

Specifically, first, the image of the subject is subjected to a checkingprocess on matching between the original images 37 and 38, 38 and 39,and 39 and 40, respectively, so that relative moving distance and movingdirection of the image of the subject are obtained. This moving distanceand moving direction are equal to the relative moving distance andmoving direction of the image pickup feasible range 29 of the videocamera 28.

Next, the one-dimensional image synthesizing unit 3 relatively moves theoriginal images 37 to 40 by the obtained moving distance in thedirection opposite to the obtained moving direction and superimposesthem. FIG. 5A and 5B is a schematic view showing the state in which theoriginal images 37 to 40 are superimposed by the method described above.Due to deviation in the direction of the arrow 33, an image 45 in thesuperimposed state has a shape partially protruding in the direction ofthe arrow 33 and in the opposite direction.

Finally, the one-dimensional image device 3 processes the image signalof the image 45 so that information other than images is not containedin the end of the image 45 in the direction of the arrow 33, andproduces the signal of the image-to-be-combined 17. As this method, forexample, when FIG. 5B is used as an example, there is cited a method inwhich like a case that a portion of the image 45 at a right side of animaginary line 46 is cut off, pixel data of pixels constituting thisportion are removed from the image signal of the image 45. Besides, inthe image signal of the image 45, data for inhibiting setting of asearch block described later may be added to pixel data expressing theportion other than images at the right side of the imaginary line 46.This is done for the purpose of preventing the search blocks from beingset at portions other than images in the combining operations of theimages-to-be-combined described later.

The one-dimensional image synthesizing unit 3 may be a so-called handscanner using a one-dimensional image pickup device in which lightreceiving areas are linearly disposed. When the image-to-be-combined 17is obtained by using this hand scanner, as shown in FIG. 6, the widthdirection of the image pickup region 13 is made almost coincident withthe arrangement direction of the light receiving areas of a hand scanner48, and the subject 11 is imaged while the hand scanner 48 is movedalong the longitudinal direction in the state where it is brought intocontact with the surface of the subject 11. At this time, since theimage pickup device moves almost in the same manner as the image pickupdevice of the video camera 28, when the hand scanner 48 is moved fromone end to the other end of the image pickup region 13 in thelongitudinal direction, a plurality of original image signals expressingoriginal images in which pixels are linearly arranged are obtained. Theone-dimensional image synthesizing unit 3 combines the original imagesignals such that these original images are sequentially arranged in theorder of imaging while the arrangement directions of the pixels are madeparallel with each other, and the image-to-be-combined 17 is obtained.

By such a method and by using the one-dimensional image synthesizingunit 3 using the video camera 28, the above-mentionedimages-to-be-combined 17 to 19 of FIG. 3 are produced. Although theshapes of the images-to-be-combined 17 to 19 and the images of thesubject in the images-to-be-combined are respectively almost similar tothe shapes of the image pickup regions 13 to 15 and the images of theirsurfaces, distortion partially occurs. It is conceivable that this isbecause, for example, the video camera 28 is manually moved by theoperator, so that it is difficult to keep the distance between thesubject 11 and the video camera 28 at a predetermined distance, and thedistance fluctuates. It is also conceivable that although the handscanner 48 is provided with a pair of rollers to make smooth movement onthe surface of the subject 11, the sliding of the pair of rollers isirregular so that the moving speed of the hand scanner 48 fluctuates.

Further, the images-to-be-combined 17 to 19 sometimes include a rotationso that the center axes of the image pickup regions 13 to 17 in thelongitudinal direction are not made parallel to each other but make someangle. It appears that this occurs from the same reason as theabove-mentioned distortion, that is, an unintentional movement of handsoccurs so that the moving direction of the video camera 28 and the handscanner 48 deviates from the original moving direction and shifts. Thus,the images of the subject in the overlapped regions 21, 22; 23, 24 donot agree with each other even if the translation of theimages-to-be-combined 17 to 19 is merely carried out, in spite of thefact that the same parts 25 and 26 of the subject are imaged.

FIG. 7 is a flowchart for explaining an image synthesizing operation inthe image synthesizing apparatus. For example, when a user of the imagesynthesizing apparatus makes instructions of combining operation, stepal proceeds to step a2.

At step a2, a plurality of signals of images-to-be-combined are inputtedin the central processing unit 5. With respect to the signals ofimages-to-be-combined, the signals obtained by the method describedbefore may be directly obtained from the one-dimensional imagesynthesizing unit 3, or desired signals among the plurality of signalsof images-to-be-combined stored in the memory 4 may be selected andread. Each time when the signals of images-to-be-combined for one pieceare inputted, step a2 proceeds to step a3.

At step a3, it is judged whether all signals of images-to-be-combinednecessary for obtaining the combined image signal are obtained. When allthe signals of the images-to-be-combined are not obtained, the processreturns to step a2, and a next signal of an image-to-be-combined isinputted. When all the signals of the images-to-be-combined areobtained, step a3 proceeds to step a4. In steps subsequent to step a4, acombining process for combining, in the signals of theimages-to-be-combined, a pair of images-to-be-combined in which theimage pickup regions are adjacent to each other and which include anoverlapped region where the same part of the subject 11 is imaged, iscarried out.

At step a4, a center block matching process for detecting an amount ofrelative translation of images of the subject in the pair ofimages-to-be-combined is carried out. Next, at step a5, a large blockmatching process for detecting an amount of detailed translation of theimages of the subject in the pair of images-to-be-combined and arelative angle of rotation is carried out. Next, at step a6, on thebasis of the detection result of the large block matching process, arotation correction process of images is applied to the pair of signalsof the images-to-be-combined. At step a7, a small block matching processfor detecting a relative distortion of images of the subject of the pairof images-to-be-combined is carried out. At step a8, on the basis of thedetection result of the small block matching process, a deformationprocess and combining process of images are applied to the pair ofsignals of the images-to-be-combined, and the pair of signals of theimages-to-be-combined are combined. The details of the processingoperations of the respective steps a4 to a8 will be described later.

At step a9, it is judged whether the series of combining processes isapplied to all the signals of the images-to-be-combined and a singlecombined image signal is obtained. If not, step a9 returns to step a4,and a series of combining processes from step a4 to step a8 is appliedto another pair of image-to-be-combined signals. Any set may be adoptedfor the set of another pair of image-to-be-combined signals as long asthe set includes an overlapped region where the same portion of thesubject 11 is imaged. For example, one of the pair ofimage-to-be-combined signals which have been combined by the methoddescribed above may be combined with another image-to-be-combinedsignal. Alternatively, a set of the image signals each obtained bycombining a pair of image-to-be-combined signals through the methoddescribed above may be adopted.

At step a9, if it is judged that all the image-to-be-combined signalsare combined, step a9 proceeds to step a10, and the process operation ofthe flowchart is ended.

A method of combining a pair of image-to-be-combined signals at step a4to step a8 in the flowchart will be described below in detail. In thefollowing explanation, a combining process of the pair ofimage-to-be-combined signals expressing the images-to-be-combined 17 and18 is used as an example, and the respective process operations will bedescribed. The images-to-be-combined 17 and 18 are produced by themethod described above, and each is a rectangular image in whichprojecting portions at both ends in the longitudinal direction are cutaway. For each of the images-to-be-combined 17 and 18, a two-dimensionalrectangular coordinate system with the origin of a vertex at the leftupper end in the drawing is set. The x and y coordinate axes in thiscoordinate system are respectively in parallel to the width directionand longitudinal direction of the images-to-be-combined 17 and 18.Hereinafter, the width direction parallel to the x-axis of coordinatesmay be called an x-direction, and the longitudinal direction parallel tothe y-axis of coordinates may be called a y-direction.

FIG. 8 is a schematic view for explaining the center block matchingprocess at step a4 of the flowchart. In the center block matchingprocess, a so-called template matching process is used, and an amount ofrelative translation of images of the subject in theimages-to-be-combined 17 and 18 is detected.

First, reference block K is set in the overlapped region 21 of theimage-to-be-combined 17. The reference block K is arranged such that itsreference point 52 is positioned on a reference axis 53 halving a sideof the image-to-be-combined 17 in the y-direction. Next, a search region55 is set in the overlapped region 22 of the image-to-be-combined 18.The search region 55 is set to include almost all region of theoverlapped region 22, and in this embodiment, it coincides with theoverlapped region 22. Like this, when the reference block K is set atthe center of the overlapped region 21 in the y-direction, even in thecase where the subjects in the images-to-be-combined 17 and 18 areshifted in any direction of the y-direction, the deviation can bedetected.

Subsequently, the so-called template matching process is carried outbetween the reference block K of the image-to-be-combined 17 and thesearch region 55 of the image-to-be-combined 18. In the process, theposition of an image most similar to the image of the subject in thereference block K is searched in the search region 55.

The template matching process will be described below in detail.

In the process operation of the process, first, a search block L whichis congruent with the reference block K is set so that a reference point52 of the block L coincides with some point in the search region 55.Next, similarity between the image of a part defined by the search blockL in the images of the subject in the search region 55 and the image ofa part defined by the reference block K in the images of the subject inthe overlapped region 21 is detected. This similarity is indicated suchthat, for example, an absolute value |k−l| of a difference value betweenpixel data k of a pixel in the reference block K and pixel data l of apixel in the search block L at the same position as the former pixel isobtained for all pixels in the reference block K, and the similarity isindicated by a total sum value d(kl) expressing the sum total. There isalso a case where it is expressed by correlation of pixel data k and lof the pixels.

When the similarity at some point is detected, a point with which thereference point 52 is made coincident is moved in the search region 55,and the similarity at the point after movement is detected by a similarmethod. Such a detecting operation of the similarity is repeatedlycarried out at each of a plurality of points in which the points aredetermined so that the ranges defined by the search blocks L cover allthe search region 55. When the detecting operations of the similarity atall the determined points are ended, the total sum values d(kl) of thesearch blocks L in which the respective points are the reference point52 are obtained, the number of which is the same as the number of thepoints. At this time, pixel data of each pixel in the search region 55is used at least one time for calculation of the total sum value d(kl).

When the total sum values d(kl) at all the points in the search region55 are detected, subsequently, among all the detected total sum valuesd(kl), a point where a minimum total sum value d(kl) is obtained, and asearch block L set with the point as reference are searched. That thetotal sum value d(kl) is minimum means that the image of the subject inthe search block L in the search region 55 is most similar to the imageof the subject in the reference block K. The search block L in which thetotal sum value d(kl) is minimum is made a minimum search block La andis shown in FIG. 8. When the minimum search block La is obtained, thetemplate matching process is ended.

The accuracy of this template matching process is improved as the sizeof the reference block K and the search block L becomes small. However,if the reference block K and the search block L are made small withoutchanging the size of the search region 55, there is increased apossibility that a search block set at a place different from thecorresponding place where a check on matching is originally made, issimilar to the reference block, so that a check on matching is made atthe quite erroneous place. It is preferable that the size of the searchand reference blocks K, L in the center block matching process is largerthan reference and search blocks f1, f2, g1, g2; ba, bb in the largeblock and small block matching processes described later. This isbecause it is sufficient if the center block matching process detectsonly an amount of translation of the subject which is broader than thedeviation of the subject to be detected in the large block and smallblock matching processes. Of course, the size of these blocks can be setarbitrarily irrespective of the relation of size described above.

In the above template matching process between the reference block K andthe respective search blocks L, in the detecting operation of thesimilarity, the plural search blocks L are directly set in the searchregion 55, and the total sum value d(kl) at each point is obtained. Inthe template matching process, a so-called layer matching process may beused.

In the layer matching process, first, the search region 55 is reduced,and the number of pixels in the search region 55 is decreased, and then,the search block L is set for the reduced region. For example, if thesearch region 55 is reduced to a half length in the x- and y-directions,respectively, four pixels of two rows and two columns of images beforereduction correspond to one pixel after reduction. By this, the numberof search blocks L to be set for covering all the search region 55becomes one fourth of the number of search blocks L to be set in thesearch region 55 before reduction. In this state, the similarity as toall search blocks L is obtained, and the position of the minimum searchblock La with the highest similarity is obtained. Next, the size of thereduced search region 55 is enlarged, and the template matching processis again applied to the images of the subject in the minimum searchblock La and in the vicinity thereof. This operation is sequentiallyrepeated until the search region 55 returns to the original size. Bythis, the process amount of the calculation process for obtaining thetotal sum value d(kl) of the respective search blocks of the entire ofthe search region 55 can be decreased.

When the images-to-be-combined 17 and 18 are superimposed so that theminimum search block La obtained by such a method and the referenceblock K are superimposed, and when the translation operation of theimages of the subject in the overlapped regions 21 and 22 is made, asshown in FIG. 9, it is understood that the images-to-be-combined 17 and18 are superimposed while being shifted in the y-direction. It ispreferable that the amount of translation of the images-to-be-combined17, 18: 18, 19 in the image synthesizing apparatus 1 of this embodimentis at most one third of the width of the image-to-be-combined in they-direction. In the following explanation, it is assumed that there islittle translation of the images of the subject of theimages-to-be-combined 17 and 18.

Like this, in the center block matching process, it is possible todetect the amount of translation of the images of the subject in thepair of images-to-be-combined in the y-direction. On the basis of thematching result of the center matching process, the central processingunit 5 makes rough positioning of the images of the subject in theimages-to-be-combined 17 and 18.

Subsequently, the large block matching process will be described belowin detail with reference to FIG. 10. The large block matching processuses a so-called template matching process and is carried out to makepositioning of images of the subject in the images-to-be-combined 17 and18 and to obtain an angle of relative rotation of theimages-to-be-combined 17 and 18.

First, reference blocks f1 and f2 are set in the overlapped region 21 ofthe image-to-be-combined 17. The set positions of the reference blocksf1 and f2 are selected so that the portions defined by the referenceblocks f1 and f2 contain distinctive portions in the image of thesubject in the overlapped region 21. The distinctive portion is aportion where the change in luminance of pixels is large, for example, aso-called edge. The position of the edge can be obtained from an imagesignal obtained by passing the image-to-be-combined signal of theimage-to-be-combined 17 through a Sobel filter. The reference blocks f1and f2 are set smaller than the reference block K of the center blockmatching process. For example, they are set to such a shape and sizethat pixels arranged in matrix of 200 rows and 60 columns are containedin the portion defined by the reference blocks f1 and f2 in theoverlapped region 21.

The separation distance W2 between the reference blocks f1 and f2 in thelongitudinal direction is a distance within the width W1 of theoverlapping portion of the images-to-be-combined 17 and 18 shown in FIG.9 in the longitudinal direction, and it is preferable that the distanceis as large as possible. Further, in order to prevent such a case thatthe deviation of the subject in the images-to-be-combined 17 and 18 inthe y-direction can not be processed, it is preferable to set thereference blocks f1 and f2 at portions except the vicinity of both endsof the overlapped region 21 in the y-direction. From these, in thefollowing explanation, it is assumed that the reference blocks f1 and f2are set at both sides of the reference axis 53 in the y-direction, andtheir reference points p1 and p2 are separated by the separationdistance W2 in parallel to the y-axis of coordinates.

Next, search regions 58 and 59 corresponding to the reference blocks f1and f2 are set in the overlapped region 22 of the image-to-be-combined18. The set position of the search regions 58 and 59 in the y-directionis selected at a position where, when the images-to-be-combined 17 and18 are arranged so that the x-axes and y-axes of coordinates become inparallel with each other, relative translation has been made, from theviewpoint of the reference block f1, along the y-direction by the amountof translation detected by the center block matching process.

The size of the search regions 58 and 59 is respectively determined bythe size of deviation of the image of the subject in theimages-to-be-combined 17 and 18 in the y-direction, the size of theoverlapped regions 20 and 21, and the change degree in the details ofthe image of the subject. For example, the width W3 of the searchregions 58 and 59 in the y-direction is shorter than the width W4 of thesearch region 55 in the y-direction at the center block matchingprocess. The width in the x-direction is almost equal to the width ofthe overlapped region 22 in the x-direction. Search blocks g1 and g2equal to the reference blocks f1 and f2 in size and shape are set in thesearch regions 58 and 59.

Subsequently, the central processing unit 5 individually performs thetemplate matching process of the reference block f1 to the search region58, and the template matching process of the reference block f2 to thesearch region 59, respectively. The process method of the templatematching process is almost equal to the center block matching process,and total sum values d1 (fg), d2 (fg) of difference values of pixelvalues are obtained as similarity as shown in the following equation.

dn(fg)=Σ|fn−gn|  (1)

The reference numerals “f” and “g” denote values of pixel dataexpressing the luminance of pixels in the reference block and the searchblock, respectively. A value of 1 or value of 2 is substituted for “n”.When a value of 1 is substituted, “f” and “g” correspond to the pixeldata of the reference block f1 and the search block g1. When a value of2 is inputted, “f” and “g” correspond to the pixel data of the pixel ofthe reference block f2 and the search block g2.

When minimum search blocks g1 a and g2 a are respectively obtained forthe search regions 58 and 59 by the template matching process, fromreference points q1 and q2 of the minimum search blocks g1 a and g2 a,an amount of detailed translation of the image of the subject in the x-and y-directions in the overlapped regions 21 and 22, and a rotationangle α are obtained.

The rotation angle α is an angle formed between a line segment z1 withboth ends of the reference points p1 and p2 of the reference regions f1and f2 and a line segment z2 with both ends of the reference points q1and q2 of the minimum search blocks g1 a and g2 a, when thecombination-subject-images 17 and 18 are arranged so that the x-axes andy-axes of coordinates become in parallel with each other. It isconceivable that the rotation angle a is the sum of an angle formedbetween they-axis of coordinates of the combination-subject-image 17 andthe line segment z1, and an angle formed between the y-axis ofcoordinates of the combination-subject-image 18 and the line segment z2.In this embodiment, since it is assumed that the line segment z1 is inparallel with the y-axis of coordinates of the combination-subject-image17, the rotation α is equal to an angle formed between the y-axis ofcoordinates of the combination-subject-image 18 and the line segment z2.

Next, a rotation correction process of an combination-subject-image willbe described below in detail. In the rotation correction process, thecoordinates of representative points corresponding to pixel values ofpixels of the images-to-be-combined 17 and 18 are relatively rotativelytransformed so that the rotation angle α is cancelled and the linesegments z1 and z2 are made parallel with each other. For example, inthis rotation transformation, the coordinates (x, y) of therepresentative point of each pixel of the image-to-be-combined 18 aretransformed with the reference point q1 of the minimum search block g1aas the center, based on the following equation. $\begin{matrix}{\begin{bmatrix}x^{*} \\y^{*}\end{bmatrix} = \quad {\begin{bmatrix}{\cos \left( {- \alpha} \right)} & {- {\sin \left( {- \alpha} \right)}} \\{\sin \left( {- \alpha} \right)} & {\cos \left( {- \alpha} \right)}\end{bmatrix}\quad\begin{bmatrix}x \\y\end{bmatrix}}} & (2)\end{matrix}$

Reference numerals “x” and “y” are values of x- and y-coordinates of therepresentative point of a pixel in the image-to-be-combined 18 beforerotation transformation. Reference numerals “x*” and “y*” are values ofx- and y-coordinates of the representative point of the same pixel afterrotation transformation. The coordinates (x, y) of the representativepoint before rotation transformation denote respectively a point in eachpixel, and for example, are coincident with so-called lattice points ofthe xy rectangular coordinate system of the image-to-be-combined 18. Therepresentative points after rotation transformation are often notcoincident with lattice points.

When such coordinate transformation is carried out, the position of therepresentative point of each pixel is angle shifted reversely when seenfrom the original image-to-be-combined 17,by the rotation angle a fromthe original position, with the reference point q1 as a basis. By this,the relative position of the image-to-be-combined 18 to theimage-to-be-combined 17 is reversely rotated by the rotation angle α sothat the images of the subject in the images-to-be-combined 17 and 18can be arranged parallel with each other.

Subsequently, the small block matching process of theimages-to-be-combined 17 and 18 will be described below in detail. Thesmall block matching process detects deviations of the subject to becorrected in the deformation of images and combining process operationdescribed later.

FIG. 11 is a flowchart for explaining a processing procedure of thesmall block matching process. FIG. 12 is a schematic view for explainingthe small block matching process. Explanation will be made withreference to both FIGS. 11 and 12. When step a6 proceeds to step a7 inthe flowchart of FIG. 7, step b1 proceeds to step b2.

At step b2, a plurality of reference blocks ba are set in the overlappedregion 21 of the image-to-be-combined 17. For example, in FIG. 12, 12reference blocks ba1 to ba12 (referred to generically as reference blockba) are prepared, and their center points are set on the line segment z1and its extension so that the respective blocks are linearly arrangedwithout overlapping. These reference blocks ba1 to ba12 are, forexample, rectangular blocks, and when the center point is made a basis,they are arranged linearly while adjacent two center points areseparated with a predetermined interval W6.

The respective reference blocks ba1 to ba12 are smaller than thereference blocks f1 and f2 of the large block matching process, and forexample, each is set to the size including pixels of 30 rows and 30columns. At this time, the reference points p1 and p2 as both ends ofthe line segment z1 are contained in the reference blocks ba2 and ba10which are arranged at second and tenth positions when counted from oneend in the y-direction. When the reference block ba is arranged, step b2proceeds to step b3.

At step b3, search blocks bb are arranged in the order described laterand are set in the overlapped region 22 of the image-to-be-combined 18.The search blocks bb are prepared individually correspondingly to thereference blocks ba and the number thereof is the same. In FIG. 12, 12search blocks bb1 to bb12 (referred to generically as search block bb)are prepared. The respective search blocks bb1 to bb12 are blocks withthe same size and shape as the reference blocks ba1 to ba12, and arerectangular blocks in this embodiment.

The search blocks bb are set so that their center points ma1 to ma12 arelinearly arranged on the line segment z2 and its extension. Among thesearch blocks bb, the search blocks bb2 and bb10 corresponding to thereference blocks ba2 and ba10 contain the reference points q1 and q2.The relative positional relation between the search blocks bb2, bb10 andthe reference points q1, q2 is equal to the relative positional relationbetween the reference blocks ba2, ba10 and the reference points p1, p2.

The search blocks bb1, bb3 to bb9, bb11, and bb12 are arranged so that,for example, when the center points are made bases, adjacent two centerpoints form an interval W7. The interval W7 is equal to a lengthobtained by dividing the length W8 of the line segment z2 by the numberof search blocks bb intervening between the reference points q1 and q2.In FIG. 12, since the number of search blocks bb between the referencepoints q1 and q2 is 8, the interval W7 is equal to a length obtained bydividing the length W8 by 8. Since the interval W7 is wider than thewidth W6 of the search block bb in the y-direction, a gap is generatedbetween adjacent two search blocks bb. On the contrary, when theinterval W7 is narrower than the width W6, there is a case where partialregions of adjacent two search blocks bb overlap with each other.

Search regions 61 respectively containing the search block bb are setaround the respective search blocks bb. For example, the center of thesearch region 61 of the search block bb1 is coincident with the centerof the search block bb1. This search region 61 is set so that forexample, the space for six pixels is added to both sides of the searchblock bb in the x- and y-directions, and its size becomes such thatpixels arranged in matrix of 42 rows and 42 columns are contained intotal. When the search blocks bb are arranged in this way, step b3proceeds to step b4. In steps subsequent to step b4, for each of thesearch regions 61 including the search blocks bb, the template matchingprocess of the respective reference blocks ba1 to ba12 to the searchregion 61 is individually carried out.

At step b4, with respect to the search region 61 to which the matchingprocess is applied, it is judged whether the change in luminance ofpixels in the search region 61 is small. For example, in a plurality ofpixels in the search region 61, when a difference between a maximumvalue and a minimum value in values of pixel data expressing theluminance is less than a predetermined difference, it is judged that thechange in luminance is small. By this judgement, it is possible topreviously remove such a search region 61 of, for example, a white imagethat the image change of images of the subject is small so that thematching process is difficult, from matching process objects. Only whenit is judged that the change in luminance of the image is large, step b4proceeds to step b5. At step b5, to the search region 61 of theobjective of the matching process which is judged as described above,the template matching process operation of the reference blocks bacorresponding to the region 61 is carried out. When the process isended, step b5 proceeds to step b6.

At step b6, it is judged whether the image of the subject in the searchregion 61 of the objective of the matching process is an image like aperiodic pattern and straight line. Specifically, for example, in thesearch region 61, when there are a plurality of points where the valueof similarity becomes higher than a judgement reference value, it isjudged that the image of the subject is the periodic pattern andstraight line. When the image of the subject is the periodic pattern andstraight line, a plurality of points in which the value of similarity isalmost the same are detected, so that there is often a case where anaccurate result can not be obtained by the template matching process.Thus, in the small block matching means, a search region containingimages of the subject including the periodic pattern and straight lineis previously removed from an subject of a process operation describedlater. Only when the image of the subject is not the periodic patternand straight line, step b6 proceeds to step b7. At step b7, from theprocessing result of the template matching process at step b5, a minimumsearch block bba is set in the search region 61.

At step b4, when it is judged that the change in luminance of images ofthe subject in the search region 61 of the matching process subject issmall, and at step b6, it is judged that the image of the subject is theperiodic pattern and straight line image, steps b4 and b6 proceed tostep b8. At step b8, the search region 61 of the objective of thematching process, and the reference block ba and the search block bbcorresponding thereto are deleted. By this, it is possible to previouslyremove one in which it is expected that the process result of thetemplate matching process becomes inferior. It is also possible to adoptsuch that after the template matching process is carried out, in boththe reference range and the search range, the total sum values of pixelvalues are detected and the difference value of these total sum valuesis obtained, and only when the difference value is not less than apredetermined value, it is judged that an accurate matching process iscarried out, and the search range is made to remain.

Like this, with respect to the search region 61 of the objective of thematching process, the process of determining the minimum search blockbba or the process of removing the search region 61 of the objective ofthe process is ended, step b7 or step b8 proceeds to step b9. At stepb9, with respect to all search regions 61 set in the overlapped region22, it is judged whether a series of process operations of step b4 tostep b8 is ended. If not, step b9 returns to step b4, and theabove-mentioned process operation is applied to the next search region61. When the template matching process is ended for all blocks, step b9proceeds to step b10, and the process operation of the flowchart isended.

FIG. 13 is a view showing the images-to-be-combined 17 and 18 after thesmall block matching process operation of FIG. 11 is applied to theimages-to-be-combined 17 and 18 of FIG. 12

As compared with the image-to-be-combined 17 of FIG. 12, in thisimage-to-be-combined 17, among reference blocks ba, as blocks notmeeting the condition of steps b4 and b6 of the flowchart, referenceblocks ba3, ba6, ba7, and ba10 are deleted. Similarly, in theimage-to-be-combined 18, the minimum search blocks bba3, bba6, bba7, andbba10 corresponding to the reference blocks ba3, ba6, ba7, and ba10 aredeleted.

Although the center points ma1, ma2, ma4, ma5, ma8, ma9, ma11, and ma12of the reference blocks ba1, ba2, ba4, ba5, ba8, ba9, ba11, and ba12remaining in the image-to-be-combined 17 exist on the line segment z1and its extension, a line segment for sequentially connecting the centerpoints mb1, mb2, mb4, mb5, mb8, mb9, mb11, and mb12 of the minimumsearch blocks bba1, bba2, bba4, bba5, bba8, bba9, bba11, and bba12remaining in the image-to-be-combined 18 becomes a polygonal line apartfrom the line segment z2 and its extension.

Next, the deformation process of images will be described below indetail. In the deformation process of image, when theimages-to-be-combined 17 and 18 are superimposed, theimage-to-be-combined 18 is geometrically transformed so that the centerpoint mak of some remaining reference block bak is coincident with thecenter point mbk of the minimum search block bbak corresponding thereto.The value of k is any one of natural numbers 1, 2, 4, 5, 8, 9, 11, and12.

In the deformation process, first, it is imagined that theimages-to-be-combined 17 and 18 are superimposed. specifically, theimages-to-be-combined 17 and 18 are superimposed such that the centerpoint ma1 of the first reference block ba1 in the overlapped region 21is coincident with the center point mb1 of the first minimum searchblock bba1 in the overlapped region 22, and the x-axes and y-axes ofcoordinates of the respective combination-subject-images 17 and 18become in parallel with each other.

FIG. 14 is a schematic view of the combination-subject-images 17 and 18superimposed by the method described above. In the drawing, anintersection point between a straight line z3 passing through the centerpoints mb1 and mb2 and a side of one end of thecombination-subject-image 18 in the y-direction is made a point mb0.Similarly, an intersection point between a straight line z4 passingthrough the center points mb11 and mb12 and a side of the other end ofthe combination-subject-image 18 in the y-direction is made a pointmb13.

From this state, the position coordinates of the representative point ofeach pixel of the combination-subject-image 18 are geometricallytransformed in a unit of a trapezoid region defined by a pair of linesegments passing through the center points mbk and mb(k+1) respectivelyand being in parallel with the x-axis of coordinates, and a line segmentwith both ends of the center points mbk and mb(k+1).

In the following, while a trapezoid region 63 is adopted as an example,the geometrical transformation method will be schematically described.In this geometrical transformation method, the trapezoid region 63 isdeformed to be coincident with a rectangle region 64, and according tothe deformation, position coordinates of each representative point aretransformed.

The trapezoid region 63 includes the center points mb1 and mb2 as itsvertexes, and a side passing through the points mb1 and mc1 is inparallel with a side passing through the points mb2 and mc2.Subsequently, a line segment and a side with certain two points as itsends, or a line segment and a side passing through certain two pointswill be denoted by continuously affixing reference numerals indicatingthe two points. The rectangular region 64 includes the center points ma1and ma2 as its vertexes and sides passing through intersection pointsmd1 and md2. The intersection points mc1, mc2; md1, md2 are intersectionpoints between straight lines passing through the center points mb1,mb2; ma1, ma2 and being in parallel with the x-axis of coordinates, anda reference axis h being in parallel with a side of one end of thecombination-subject-image 18 in the x-direction. The trapezoid region 63and the rectangular region 64 are shown by oblique lines high on theright and low on the right in FIG. 14.

First, from FIG. 14, it is understood that the width W11 of thetrapezoid region 63 in the y-direction is wider than the width W12 ofthe rectangular region 64 in they-direction. Thus, the trapezoid region63 is compressed and deformed at a constant transformation ratio alongthe x-direction with respect to the y-direction. This transformationratio is expressed by the following equation.

(transformation ratio in the y-direction)=W12/W11  (3)

By this, translation is made so that the sides mb1mc1 and mb2mc2 of thetrapezoid region 63 coincide with the sides ma1md1 and ma2md2 of therectangular region 64.

Moreover, from FIG. 14, it is understood that the side mb1mb2 of thetrapezoid region 63 and the side ma1ma2 of the rectangular region 64 aredifferent in length of a side and in angle to the y-axis of coordinates.Thus, with the reference axis h being in parallel with the y-axis ofcoordinates as the basis, the trapezoid region 63 is enlarged anddeformed at a transformation ratio continuously changing along they-direction, with respect to the x-direction. For example, thetransformation ratio in the x-direction on an imaginary line 65 being inparallel with the x-axis of coordinates and passing through a portionbetween the center points ma1 and ma2 is expressed by the followingequation.

(transformation ratio in the x-direction)=W13/W14  (4)

W13 denotes a distance between the reference axis h and the line segmentwith the center points mb1 and mb2 as its both ends. W14 denotes adistance between the reference axis h and the straight line z3 on theimaginary line 65. Like this, the transformation ratio of a region inthe x-direction is determined for each imaginary line set at apredetermined interval shorter than the interval between the centerpoints ma1 and ma2. By this, rotation and movement are made so that thesides mb1mb2 and mc1 mc2 of the trapezoid region 63 coincide with thesides ma1ma2 and md1md2 of the rectangular region 64.

Like this, when the trapezoid region 63 is deformed in trapezoid, thesides mb1mb2, mb1mc1, and mb2mc2 of the trapezoid region 63 aftertransformation coincide with the sides ma1ma2, ma1md1, and ma2md2 of therectangular region 64. In FIG. 14, the right side of the trapezoidregion 63 expands as shown by a phantom line. The coordinates of therepresentative point of each pixel in the trapezoid region 63 aretransformed so that an interval between two adjacent representativepoints in the x- and y-directions is reduced or enlarged at thetransformation ratio in the x- and y-directions. Thus, the position ofeach representative point in the x- and direction is sequentially movedby the amount of transformation of the interval.

Subsequently, a trapezoid region with another center point mb as itsvertex is sequentially deformed by a similar method to the geometricaldeformation method of the trapezoid region 63 described above.Specifically, the respective trapezoid regions are geometricallydeformed so that the center point mbk coincides with the center pointmak. Since points corresponding to the points mb0 and mb13 can not bedetermined in a trapezoid region containing the points mb0 and mb1 asits vertexes, and in a trapezoid region containing the points mb12 andmb13 as its vertexes, a transformation ratio in the y-direction can notbe determined by the method described above. Thus, the transformingprocess is carried out on the assumption that the transformation ratio,in the y-direction, of the rectangular region containing the point mb0and mb1 as its vertexes, and the trapezoid region containing the pointsmb12 and mb13 as its vertexes is equal to the transformation ratio, inthe y-direction, of the trapezoid region 63 and the trapezoid regioncontaining the points mb11 and mb12 as its vertexes.

The reference axis h may be set at any place as long as it is in thecombination-subject-image 18 and is in a region in the direction goingaway from the overlapped region 22 beyond the center point mb. Thereference axis h coincides with, for example, the side of one end of thecombination-subject-image 18 at the side of the overlapped region 23.Further, it may coincide with the center axis of thecombination-subject-image 18 in the x-direction. Moreover, it maycoincide with an imaginary line passing through the center points of allreference blocks ba at the small block matching process when thecombination-subject-images 18 and 19 are objective of the combiningprocess.

When the reference axis is made coincident with the imaginary linepassing through the center points of the reference blocks ba, in thegeometrical transformation process of images, the transformation ratioof the image-to-be-combined 18 in the x-direction is determined on thebasis of the distance between the reference axis h and the side ma1ma2,so that pixels on the reference axis h are linearly arranged on the samereference axis h even after the geometrical transformation. From this,it is understood that after image combining of the images-to-be-combined17 and 18 is carried out, when image combining of the combined image andthe image-to-be-combined 19 is carried out, the calculation process ofthe small block matching process in the process of the latter imagecombining becomes easy.

By such a method, the position coordinates of the representative pointof each pixel of the image-to-be-combined 18 are transformed accordingto the image-to-be-combined 17. A superimposed image in which theimage-to-be-combined 18 after transformation and the originalimage-to-be-combined are superimposed so that the center points ma andmb coincide with each other, is made a basis of a combined image. Whenthe superimposed image is produced, a portion, in theimage-to-be-combined 18, closer to the center of theimage-to-be-combined 17 than the imaginary line passing through thecenter point mb of the image-to-be-combined 18, that is, a left portionto the imaginary line in FIG. 14 is cut off. The imaginary line isexpressed by a thick solid line in FIG. 14 before image deformation, andcoincides with a combining reference axis H after image deformation.This process is equivalent to deletion of pixel values of pixelsconstituting the part, in the image-to-be-combined signals of theimage-to-be-combined 18. Thus, in the superimposed image, only the endat the center side of the image-to-be-combined 18 from the combiningreference axis H passing through all the center points ma of theimage-to-be-combined 17 overlaps with the end of theimage-to-be-combined 18. Among superimposed images, a portion where theimages-to-be-combined 17 and 18 superimpose with each other, will behereinafter referred to as an overlapped portion.

The representative points of pixels constituting the superimposed imageare often deviated from lattice points of the xy rectangular coordinatesystem set for the superimposed image. The combined image is equal tothe superimposed image in the coordinate axis of the xy rectangularcoordinate system, and is an image formed of pixels with the latticepoints of the coordinate system as representative points. The pixelvalue of a pixel of the combined image is determined on the basis of thefollowing determining method. The determining operation of the pixelvalue is carried out at the same time as the geometrical transformationoperation of the image.

First, among all pixels of the combined image, it is assumed that thepixel values of representative points of remaining pixels other than theoverlapped portion are equal to pixel values at lattice points in thesuperimposed image at the same position coordinates as the positioncoordinates of the representative points. For example, when therepresentative point of a pixel is coincident with the lattice point,the pixel value corresponding to this representative point is used as itis. When the lattice point does not coincide with the representativepoint of the superimposed image, the pixel value of the lattice point isinterpolated and determined from pixel values of representative pointsof a plurality of pixels near the lattice point in the superimposedimage by using a so-called linear interpolation method. In the linearinterpolation method, a mean value of values obtained by multiplyingpixel values of representative points of pixels at four points aroundthe lattice point by a coefficient according to the distance between thelattice point and the respective representative points is made a pixelvalue of the lattice point.

The pixel value of the representative point of the pixel of theoverlapped portion is determined by using the linear interpolationmethod from the weighted average value of pixel values of representativepoints of pixels of the superimposed image near the lattice point in thesuperimposed image at the same position coordinates as the positioncoordinates of the representative point. The coefficient value of theweighted average value is determined based on the fact to which of theimages-to-be-combined 17 and 18 the pixel of the superimposed imageoriginally belongs, and based on the distance between the representativepoint of the pixel and the combining reference axis H.

FIG. 15 is a graph showing the relation between the coefficient value ofthe weighted average value of the pixel values and the x-coordinatevalue of the superimposed image. It is assumed that the origin of thesuperimposed image and the combined image in the xy rectangularcoordinate system coincides with, for example, the right upper vertex ofthe image, that is, the origin of the image-to-be-combined 17, and thex-coordinate value increases as it approaches the image-to-be-combined18. The coefficient value is a value between a value of 0 and a value of1 according to the x-coordinate value of the superimposed image.

The coefficient value of pixels belonging to the image-to-be-combined 17is shown by a polygonal line 71. This coefficient value keeps a value of1 between the origin and the combining reference axis H. The portionbetween these is a portion constituted by only pixels belonging to theimage-to-be-combined 17 in the superimposed image, and the combiningreference axis H is a boundary between the portion and the superimposedportion. The coefficient value becomes small as the position goes awayfrom the combining reference axis in the direction where thex-coordinate value increases, and the coefficient value becomes a valueof 0 at the point separate from the combining reference axis H by thepredetermined distance W16. Even if the x-coordinate value is furtherincreased, the coefficient value keeps a value of 0. The predetermineddistance W16 is set smaller than the width of the overlapped portion inthe x-direction, and for example, it is equal to the length in which 15pixels of the superimposed image are arranged in the x-direction.

The pixel belonging to the image-to-be-combined 18 is shown by apolygonal line 72. This coefficient value keeps a value of 0 between theorigin and the combining reference axis H. The coefficient value becomeslarge as the position goes away from the combining reference axis H inthe direction where the x-coordinate value is increased, and thecoefficient value becomes a value of 1 at the point separate from thecombining reference axis H by the predetermined distance W16. Even ifthe x-coordinate value is further increased, the coefficient value keepsa value of 1. Since the absolute values of change rates of coefficientvalues near the combining reference axis H of the images-to-be-combined17 and 18 are equal to each other, the coefficient values of the pixelvalues of the images-to-be-combined 17 and 18 at the point separate fromthe combining reference axis H by half of the distance W16 are equal toeach other.

Thus, in the determining operation of pixel values of the superimposedportion, the linear interpolation is carried out by using a valueobtained by multiplying the pixel values of each pixel of thesuperimposed portion by the coefficient value of the weighted average.By this, even when the luminance in the entire of the images of thesubject in the images-to-be-combined 17 and 18 is different, the imagescan be combined so that the change in luminance is made smooth at thesuperimposed portion of the images.

By using the method as described above, the images-to-be-combined 17 and18 can be combined. When the image-to-be-combined 19 is further combinedwith the images-to-be-combined 17 and 18, the combined image and theimage-to-be-combined 19 are combined by the image combining methoddescribed above, so that the combined image in which theimages-to-be-combined 17 to 19 are combined is obtained. As the imagecombining method of the images-to-be-combined 17 to 19, matchingprocesses to the images-to-be-combined 17, 18; 18, 19 are individuallycarried out, and then, the image deformation process of theimages-to-be-combined 17 to 19 may be carried out. In this imagedeformation process, image combining is made first for either one of thesets of the images-to-be-combined 17, 18; 18, 19, and the combined imageis combined with the remaining image-to-be-combined. When the remainingimage-to-be-combined and the combined image are combined, the deviationof position of the reference block ma in the image combining process isadded, as an offset, to the positional relation between the center pointma of the reference block ba and the center point mb of the minimumsearch block bba at the small block matching process. By this, imagecombining can be made by the same image deformation method as the methoddescribed above.

By such an image combining operation, it is possible to obtain acombined image with the number of pixels larger than theimage-to-be-combined. When the size of the pixel is made coincident withthe pixel of the image-to-be-combined, the combined image becomes animage with a wider angle of view and a wider range than theimage-to-be-combined. When the size of the image is made coincident withthe image-to-be-combined, it becomes an image with higher resolutionthan the image-to-be-combined.

In the image synthesizing apparatus, although a gray scale image whichis a white and black image is the image-to-be-combined, a color imagemay be used as the image-to-be-combined. At this time, as pixel valuesof an image combining method, for example, the value of a luminancesignal for each pixel is calculated and is used. When the values ofprimary color signals indicating the luminance of primary colors lightof red, blue and green are individually obtained for each pixel, onlythe value of either one primary color signal is made the pixel value,and the image combining method is carried out for only the primary colorsignal. The remaining primary color signals are transformed on the basisof the processing result of the image combining process of either oneprimary color signal. As the either one primary color signal, it ispreferable to use a green primary color signal in which contribution tothe luminance signal is maximum.

Further, in the image synthesizing apparatus of this embodiment,although three images-to-be-combined are used, the number ofimages-to-be-combined is not limited to this, but may be furtherincreased. Although the arrangement of the images-to-be-combined is onlyin one direction, if each image-to-be-combined includes an overlappedregion overlapping with at least an another image-to-be-combined, theymay be arranged in matrix in two directions crossing at right angles.Further, if the image-to-be-combined includes an overlapped region withanother image-to-be-combined, the original image may be used as it is. Aone-dimensional image synthesizing unit for obtaining thisimage-to-be-combined is not limited to the video camera 28, but acopying machine or a scanner may be used. The image-to-be-combined maybe an image which is inputted as image signals and is synthesized by apersonal computer.

Further, the method of the image combining process is not limited, butanother method may be used as long as the same result can be obtained.For example, the matching method of each block matching process may be amatching process other than the template matching process. The referenceblock and search block may have a shape other than a rectangular block.

INDUSTRIAL APPLICABILITY

According to the present invention, an image synthesizing apparatuscombines a plurality of images-to-be-combined overlapping with eachother and obtains a single combined image. In the image combiningprocess operation, after the deviations of images of the subject in therespective images-to-be-combined are detected by two-step matchingprocesses, movement of relative position and geometrical transformationof images are carried out to cancel the deviation for each of theimages-to-be-combined, and the respective images-to-be-combined arecombined. By this, it is possible to prevent deformation of the image ofthe subject in the combined image by accumulation of deviations of theimages-to-be-combined. Since the matching process is carried out in twosteps, the number of matching processes can be decreased.

Further, according to the invention, in the first matching means, first,in the first matching process, matching for obtaining relativepositional deviations of the images of the subject near the center inthe overlapped region of a pair of images-to-be-combined is obtained.Next, on the basis of the result of the first matching, in the secondmatching process, two pairs of reference regions and search regions areset, and matching for detecting detailed corresponding points in theimages is obtained for each pair. By this, even when the state ofdeviation of images of the subject in a pair of images-to-be-combined isindefinite, the reference regions and search regions of the secondmatching process can be arranged at the optimum position for detectingthe parallel deviation of images of the subject in the overlappedregion. By using the two pairs of regions, the size and the deviation ofrotation of the image of the subject of a pair of images-to-be-combinedcan be detected at the same time. Further, since the size of thereference region in the matching process is set so that the region atthe second is smaller than the region at the first, rough detection ofdeviation of images and detection of detailed corresponding points inthe images can be carried out respectively by an optimum method.

Further, according to the invention, the image synthesizing apparatusperforms rotation transformation of representative points of pixels foreach of adjacent set images-to-be-combined of matching objects duringthe process by the second matching means. By this, the overlapped regionof the image-to-be-combined and its near portion can be combined so thatthe images of the subject are naturally connected in the image aftercombining.

Further, according to the invention, the second matching means uses apredetermined index, and uses only matching obtained in the searchregion in which the image of the subject is eligible as a comparativeobjective, to obtain a transformation amount of geometricaltransformation. Further, according to the invention, the eligibility isdetermined by a difference value between total sum values of pixelvalues in the reference and search ranges, a difference value betweenthe maximum value and minimum value of pixel values in the searchregion, and presence or absence of periodicity of the images of thesubject. By this, it is possible to prevent distortion of images of thesubject from being increased by carrying out a combining process witherroneous matching. Thus, it is possible to improve the picture qualityof the combined image.

Further, according to the invention, at image combining, the respectiveimages-to-be-combined are partially geometrically transformed into atrapezoid shape with a transformation reference axis as a basis. Bythis, especially when not less than three images-to-be-combined arecombined, it is possible to prevent accumulation of deviations of imagesof the subject in the images-to-be-combined. Thus, the picture qualityof combining images can be improved.

Further, according to the invention, since the reference region of thesecond matching process is set on the transformation reference axis,especially when not less than three images-to-be-combined are combined,it is possible to decrease a calculation process amount of combiningprocess. Thus, the load of image combining means can be reduced.

Further, according to the invention, pixel values of pixels in a portioncorresponding to the overlapped region of the image-to-be-combined inthe combined image are obtained through weighted average of pixels ineach overlapped region of a pair of images-to-be-combined. By this, evenwhen the luminance of each of a plurality of images-to-be-combined isdifferent from one another, it is possible to make a joint of combinedimages smooth. By this, the picture quality of the combined image can beimproved.

What is claimed is:
 1. An image synthesizing apparatus comprising:image-to-be-combined producing means for producing a plurality ofsignals of images-to-be-combined, the signals representing theimages-to-be-combined including different portions of a subject, themeans producing signals so that an image-to-be-combined including aportion of the subject and an adjacent image-to-be-combined including aportion adjacent to said portion of the subject have an overlappedregion where a same portion of the subject is imaged in theimages-to-be-combined; first matching means for making a check onmatching of the images of the subject in the overlapped region for eachset of adjacent images-to-be-combined among the image-to-be-combinedsignals produced by the image-to-be-combined producing means, anddetecting a relative positional deviation between the overlapped regionson the basis of a result of the check on matching; second matching meansfor setting a plurality of reference regions of a predetermined size inthe overlapped region of one of the images-to-be-combined whose relativepositional deviation is detected by the first matching means, forsetting a search region larger than the reference region for each of thereference regions in the overlapped region of the other of theimages-to-be-combined on the basis of the relative positional deviation,for making a check on matching of the images of the subject between thereference region and the search region corresponding to the referenceregion, and for detecting a relative positional deviation between theimages of the subject on the basis of a result of the check on matching;and image combining means for combining all the image-to-be-combinedsignals to superimpose the reference region and the search regioncorresponding to the reference region for each set of adjacentimages-to-be-combined while partially deforming the respectiveimages-to-be-combined to cancel the relative positional deviationdetected by the second matching means, and producing combined imagesignals expressing a single combined image with respect to the subject.2. The image synthesizing apparatus of claim 1, wherein the firstmatching means: firstly, sets a first reference region of apredetermined size at a predetermined position in an overlapping area ofone of the set of the adjacent images-to-be-combined, making a check onmatching of the images of the subject between the overlapped region ofthe other image-to-be-combined and the first reference region, anddetects a relative positional deviation of the images of the subject onthe basis of the obtained matching; next, sets at least two secondreference regions smaller than the first reference region in parallelwith the detected deviation direction in the one of the overlappedregions, and sets a second search region smaller than the overlappedregion in the overlapped region of the other subject region for each ofthe second reference regions on the basis of the detected relativepositional deviation; and makes a check on matching of the images of thesubject between the second search region and the second reference regioncorresponding to the second search region, and detects the relativepositional deviation between the overlapped regions on the basis of anobtained result of the check on matching.
 3. The image synthesizingapparatus of claim 1, further comprising: rotation transformation meansfor obtaining a relative rotation angle between the adjacentimages-to-be-combined for each set of the adjacent images-to-be-combinedon the basis of the relative positional deviation between the overlappedregions detected by the first matching means, and for making rotationtransformation of the image-to-be-combined signals so that therespective images-to-be-combined undergo relative angular displacementin the direction where the obtained relative rotation angle iscancelled, wherein the second matching means makes a check on matchingof the search region for set of the images-to-be-combined in which therelative position is rotatively transformed by the rotationtransformation means.
 4. The image synthesizing means of claim 1,wherein the second matching means gives to the image synthesizing meansonly a result of the check on matching for the search region in whichthe image of the subject meets a predetermined index of eligibility as acomparative objective, among results of the check on matching obtainedfor the respective plurality of search regions.
 5. The imagesynthesizing apparatus of claim 4, wherein the images-to-be-combined isconstituted by a plurality of pixels, the reference region is equal toor larger than a single pixel in size, the result of the check onmatching to the respective search regions is expressed in a position ofthe search range which is equal to the reference region in size andshape in the respective search regions, and in which a total sum valueof pixel values expressing luminance of all pixels in the search rangeis closest to a total sum of pixel values of the reference region, andit indicates eligibility of the objectives as a comparative objectivethat a difference value between the total sum value of the pixel valuesin the search range of the check on matching and the total sum value ofthe pixel values of the reference region is less than a predeterminedreference value.
 6. The image synthesizing apparatus of claim 4, whereineach of the images-to-be-combined is constituted by a plurality ofpixels, the reference region is equal to or larger than a single pixelin size; the result of the check on matching to the respective searchregions is expressed in a position of a search range which is equal tothe reference region in size and shape in the respective search regions,and in which a total sum value of pixel values expressing luminance ofall pixels in the search range is closest to a total sum value of pixelvalues of the reference region, and it indicates eligibly of theobjective as a comparative objective that a difference value between amaximum value and a minimum value among pixel values in the search rangeof the matching is not less than a predetermined reference value.
 7. Theimage synthesizing apparatus of claim 4, wherein it indicateseligibility of the objectives as a comparative objective that in theimages of the subject in the search region, portions with the samechange in luminance and the same shape are not periodically repeatedplural times.
 8. The image synthesizing apparatus of claim 1, whereinthe image-to-be-combined is formed of a plurality of pixels arranged ina matrix form along predetermined one and another direction, and in thegeometrical transformation, for each of the images-to-be-combined, adistance between a predetermined transformation reference axissubstantially in parallel with the one direction and each of the pixelsis enlarged or reduced in the overlapped region, and a mutual distancein a pixel group composed of a plurality of pixels arranged linearlyalong the other direction is enlarged or reduced, so that a pixelarrangement is transformed.
 9. The image synthesizing apparatus of claim8, wherein the transformation reference axis passes through the centersof the respective reference regions of the second matching means. 10.The image synthesizing apparatus of claim 1, wherein theimage-to-be-combined and the combined image are respectively composed ofa plurality of pixels, and among all pixels of the combined image, apixel value expressing luminance of each pixel of a portioncorresponding to the overlapped region of the image-to-be-combined is aweighted average value obtained by attaching a weight, which isdetermined corresponding to a distance from a predetermined combiningreference axis in the overlapped region, to a pixel value of each pixelof two overlapped regions of the image-to-be-combined corresponding tothe pixel.
 11. An image synthesizing method comprising the steps of:producing a plurality of signals of images-to-be-combined, the signalsrepresenting the images-to-be-combined including different portions of asubject, so that an image-to-be-combined including a portion of thesubject and an adjacent image-to-be-combined including a portionadjacent to said portion of the subject have an overlapped region wherea same portion of the subject is imaged in the images-to-be-combined;making a check on matching of the images of the subject in theoverlapped region for each set of adjacent images-to-be-combined amongthe produced image-to-be-combined signals, and detecting a relativepositional deviation between the overlapped regions on the basis of aresult of the check on matching; setting a plurality of referenceregions of a predetermined size in the overlapped region of one of theimages-to-be-combined whose relative positional deviation is detected,for setting a search region larger than the reference region for each ofthe reference regions in the overlapped region of the other of theimages-to-be-combined on the basis of the relative positional deviation,for making a check on matching of the images of the subject between thereference region and the search region corresponding to the referenceregion, and for detecting a relative positional deviation between theimages of the subject on the basis of a result of the check on matching;and combining all the image-to-be-combined signals to superimpose thereference region and the search region corresponding to the referenceregion for each set of adjacent images-to-be-combined while partiallydeforming the respective images-to-be-combined to cancel the detectedrelative positional deviation, and producing combined image signalsexpressing a single combined image with respect to the subject.